Video conference system and method for performing the same

ABSTRACT

A video conference system comprising: a control device located at each of the at least two conference sites, the control device being connected to the video camera and the display device; and a server connected to the control device via a communication network, the control device comprising: a control terminal configured to control the display device; and an image display format selecting unit connected to the control terminal, wherein the image display format includes at least one format in which a predetermined process is performed to image data of the video image, the control device controls transmission and reception of the image data, and the display device located at a site displays the video image of another site with the selected image display format.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to and incorporates by referencethe entire contents of Japanese Patent Application No. 2012-203698 filedin Japan on Sep. 14, 2012, Japanese Patent Application No. 2013-048503filed in Japan on Mar. 11, 2013, and Japanese Patent Application No.2013-140201 filed in Japan on Jul. 3, 2013.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a conference device installed in eachof locations to conduct a video conference between the locations and aserver that has a communication connection with the conference devices.

2. Description of the Related Art

In recent years, a video conference system has become widely used thatenables a remote conference (video conference) by connecting terminaldevices (conference devices) installed in remote places (locations) viaa network such as the Internet. The conference devices are installed inconference rooms or the like in the respective locations and exchangeimages and voice of conferees with the other conference devices toconduct the video conference. Specifically, each of the conferencedevices captures images of the conferees participating in the videoconference by a camera and collects voice of the conferees by amicrophone, transmits the image data and the voice data to the otherconference devices, receives image data and voice data from the otherconference devices, displays a conference screen using the receivedimage data on a display unit, and outputs the voice data, as audio, froma speaker.

Incidentally, if a general camera is used as the camera of theconference device, it may be difficult to capture an image of the entireconference room because the angle of view of the general camera isnarrow. In particular, when a large number of conferees participate inthe conference, in some cases, it may be difficult to capture all of theconferees within the field of view of the camera. To cope with this,there is a known technology to provide a wide-angle lens in the cameraand perform image processing on a wide-angle image to zoom in or pan thescene of the conference (see Japanese Patent Application Laid-open No.H08-279999).

In the video conference system as described above, there is a demand bythe conferees to conduct the video conference while viewing a desiredconference screen according to the discussions in the conference oraccording to the progress of the conference. For example, there is ademand to conduct the conference while viewing the situation in theother conference rooms (situation of all of the conferees) or focusingon a specific conferee.

However, the technology disclosed in Japanese Patent ApplicationLaid-open No. H08-27999 only enables to generate image data of theentire conference room or image data of an enlarged part of theconference room by performing image processing on the wide-angle image,but is unable to display a conference screen by using image data thatthe conferees want to view. Therefore, for example, even when theconferees want to view the situation of all of the conferees in theother locations, a conference screen with a close-up view of a specificconferee, such as a speaker, may be displayed. Namely, the conferencescreen desired by the conferee and the actual conference screen do notalways match each other.

Therefore, there is a need for a conference device and a server capableof displaying a conference screen as desired by a conferee.

SUMMARY OF THE INVENTION

It is an object of the present invention to at least partially solve theproblems in the conventional technology.

According to an aspect of the invention, a video conference system forholding a conference between at least two conference sites by displayinga video image that is captured with a video camera on a display devicelocated at the conference site is provided. The video conference systemincludes: a control device located at each of the at least twoconference sites, the control device being connected to the video cameraand the display device; and a server connected to the control device viaa communication network, the control device includes: a control terminalconfigured to control the display device; and an image display formatselecting unit connected to the control terminal, wherein the imagedisplay format includes at least one format in which a predeterminedprocess is performed to image data of the video image that is capturedby the video camera, the control device controls transmission andreception of the image data, and the display device located at a site inwhich the image display format is selected by the image display formatselection unit displays the video image of another site to which theselected image display format is noticed, with the selected imagedisplay format.

According to another aspect of the invention, a method for performing avideo conference system for holding a conference between at least twoconference sites by displaying a video image that is captured with avideo camera on a display device located at the conference site isperformed. The video conference system includes: a control devicelocated at each of the at least two conference sites, the control devicebeing connected to the video camera and the display device; and a serverconnected to the control device via a communication network, the controldevice includes: a control terminal configured to control the displaydevice; and an image display format selecting unit connected to thecontrol terminal, wherein the image display format includes at least oneformat in which a predetermined process is performed to image data ofthe video image that is captured by the video camera, the control devicecontrols transmission and reception of the image data, the displaydevice located at a site in which the image display format is selectedby the image display format selection unit displays the video image ofanother site to which the selected image display format is noticed, withthe selected image display format. The method includes: selecting adesired image display format, by the image display format selection unitlocated at the site in which the image display format is selected by theimage display format selection unit; noticing the selected image displayformat to another control terminal located at the other site to whichthe selected image display format is noticed; performing on the imagedata a process according to the selected image display format; receivingfrom the server the image data with the selected image display format;and displaying the video image on the display device located at the sitein which the image display format is selected by the image displayformat selection unit.

The above and other objects, features, advantages and technical andindustrial significance of this invention will be better understood byreading the following detailed description of presently preferredembodiments of the invention, when considered in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of the entireconfiguration of a video conference system;

FIG. 2A is a block diagram illustrating a hardware configuration of aserver according to a first embodiment;

FIG. 2B is a block diagram illustrating a configuration example of maincomponents inside a conference device;

FIG. 3A is a diagram illustrating an example of a field of view of acamera during a video conference;

FIG. 3B is a diagram illustrating image data of the field of viewillustrated in FIG. 3A;

FIG. 4 is a diagram illustrating an example of overall image data;

FIG. 5 is a diagram illustrating an example of partially-enlarged imagedata;

FIG. 6 is a flowchart illustrating the flow of an image datatransmission process;

FIG. 7 is a flowchart illustrating the flow of an image data receptionprocess;

FIG. 8 is a diagram illustrating the flow of data transmitted andreceived between the server and the conference devices according to thefirst embodiment;

FIG. 9 is a diagram illustrating another flow of data transmitted andreceived between the server and the conference devices according to thefirst embodiment;

FIG. 10 is a diagram illustrating the flow of data transmitted andreceived between the server and the conference devices according to afirst modification;

FIG. 11 is a diagram illustrating another flow of data transmitted andreceived between the server and the conference devices according to thefirst modification;

FIG. 12 is a flowchart illustrating the flow of a received-image displayformat switching process;

FIG. 13 is a flowchart illustrating the flow of an image data transferprocess;

FIG. 14 is a flowchart illustrating the flow of an image datatransmission process according to a third modification;

FIG. 15 is a diagram illustrating the flow of data transmitted andreceived between the server and the conference devices according to thethird modification;

FIG. 16 is a flowchart illustrating the flow of an image datatransmission process according to a fourth modification;

FIG. 17 is a flowchart illustrating the flow of image processingperformed by a server according to the fourth modification;

FIG. 18 is a diagram illustrating the flow of data transmitted andreceived between the server and the conference devices according to thefourth modification;

FIG. 19 is a block diagram illustrating a functional configuration of aninformation processing apparatus according to a second embodiment;

FIG. 20 is a flowchart illustrating the flow of video conference controlperformed by an information processing apparatus serving as acaptured-image transmission side;

FIG. 21 is a flowchart illustrating the flow of video conference controlperformed by an information processing apparatus serving as acaptured-image reception side;

FIG. 22 is a sequence diagram illustrating the flow of data in a displaycontrol system according to the second embodiment;

FIG. 23 is a sequence diagram illustrating the flow of data in thedisplay control system according to the second embodiment; and

FIG. 24 is a flowchart illustrating the flow of video conference controlperformed by the information processing apparatus serving as thecaptured-image transmission side when receiving processing informationindicating a changed image processing method.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Exemplary embodiments of the present invention will be explained indetail below with reference to the accompanying drawings. The presentinvention is not limited to the embodiments below. In the drawings, samecomponents are denoted by the same reference numerals and symbols.

First Embodiment

FIG. 1 is a block diagram illustrating an example of the entireconfiguration of a video conference system 1 according to a firstembodiment. As illustrated in FIG. 1, the video conference system 1includes a server 3 and a plurality of conference devices 5 (5-1, 5-2,5-3, 5-4, . . . ), which are connected to one another via a network N,such as the Internet. As the server 3 and the conference devices 5,general-purpose computers, such as server computers, workstations, orpersonal computers, may be used.

The server 3 performs a process of monitoring whether a communicationconnection is established with each of the conference devices 5, aprocess of calling the conference devices 5 installed in locations(participating locations) participating in a video conference at thebeginning of the conference, or a process of transferring, to otherparties (other participating locations), image data or voice data thatis transmitted by the conference devices 5 in the participatinglocations with which the communication connections have been establishedin response to a call during the video conference.

Each of the conference devices 5 is installed in a conference room orthe like in a location in a remote place, and operated by a conferee ofthe video conference. During the video conference, the conference device5 in each of the participating locations transmits image data of theconferee captured by a camera 51 (to be described later) and voice dataof the conferee collected by a microphone 52 (to be described later) tothe server 3, receives image data and voice data that are transmitted bythe conference devices 5 in the other participating locations and thatare transferred by the server 3, and outputs the received data bydisplaying the image data as a conference screen on a display unit 54and outputting the voice data, as audio, from a speaker 55.

For example, when the three conference devices 5-1 to 5-3 of the videoconference system 1 illustrated in FIG. 1 participate in a videoconference, image data and voice data transmitted by the conferencedevice 5-1 are transferred to the conference devices 5-2 and 5-3 servingas the other parties under the control of the server 3, but are nottransferred to the conference device 5-4. Similarly, image data andvoice data transmitted by each of the conference devices 5-2 and 5-3 aretransferred to the conference devices 5-1 and 5-3 or to the conferencedevices 5-1 and 5-2 serving as the other parties under the control ofthe server 3, but are not transferred to the conference device 5-4. Asdescribed above, in the video conference system 1, a video conference isconducted between the participating locations where two or moreconference devices 5 that have established the communication connectionswith the server 3 are installed. In the following, the conference device5 in each of the participating locations participating in a single videoconference through the communication connection established with theserver 3 may be referred to as “the connected device 5” if needed.

FIG. 2A is a block diagram illustrating a hardware configuration of theserver 3 according to the first embodiment. The server 3 according tothe first embodiment includes a display unit 30, a central processingunit (CPU) 31, a recording device 32, a memory 33, an operating unit 34,and a local area network (LAN) interface (I/F) unit 35.

The display unit 30 includes, for example, a liquid crystal display(LCD), an electroluminescence (EL) display, or a cathode ray tube (CRT)display, and displays various types of information input by the CPU 31.

The recording device 32 includes, for example, a nonvolatile memory or ahard disk drive (HDD), and stores therein programs to be executed by theCPU 31 to control the units of the server 3 or to control transfer ofcaptured images and voice when a video conference is conducted.

The memory 33 is used as a working memory of the CPU 31, and has amemory area for loading the programs executed by the CPU 31 andtemporarily storing data or the like used by the CPU 31 during executionof the programs.

The operating unit 34 includes a keyboard, a mouse, a touch panel,various switches, or the like, and is used by a user (for example, anadministrator of the server 3) to input operation. The operating unit 34notifies the CPU 31 of the input operation.

The LAN I/F unit 35 is connected to an external device (for example, theconference device 5 installed in each location) via the network N, suchas the Internet, and capable of transferring captured images and voicetransmitted and received between the conference devices 5. In the firstembodiment, the LAN I/F unit 35 may be a wired LAN connected to theEthernet (registered trademark) compliant with 10 Base-T, 100 Base-TX,or 1000 Base-T, a wireless LAN compliant with 802.11a/b/g/n, or thelike.

The CPU 31 is a control unit that controls transfer of the capturedimages and voice when controlling the units of the server 3 or when avideo conference is conducted.

FIG. 2B is a block diagram illustrating a configuration example of maincomponents inside the conference device 5. As illustrated in FIG. 2B,the conference device 5 includes the camera 51, the microphone 52, anoperating unit 53, the display unit 54, the speaker 55, a recordingdevice 56, a memory 57, a LAN I/F unit 58, and a CPU 59 serving as atransmission format designating means, an image transmitting means, areception format designating means, and an image display means.

The camera 51 is used to input an image of conferees and configured tocontinually capture an image of the situation inside a conference roomand output generated image data to the CPU 59. The camera 51 includes awide-angle lens and is installed, before a start of the videoconference, at an appropriate position inside the conference room suchthat all of the conferees can be captured within the field of view (theangle of view). The microphone 52 is used to input voice of theconferees and configured to continually output collected voice data ofthe conferees to the CPU 59.

The operating unit 53 includes an input device, such as a keyboard, amouse, a touch panel, or various switches, and outputs input datacorresponding to input operation to the CPU 59.

The display unit 54 includes a display device, such as an LCD, an ELdisplay, or a CRT display, and displays or outputs various screens, suchas a conference screen for displaying image data (processed image datato be described later) input by the CPU 59. The speaker 55 outputs, asaudio, voice data input by the CPU 59.

The recording device 56 is used to record programs for operating theconference device 5 to implement various functions of the conferencedevice 5, data to be used during execution of the programs, or the like.The recording device 56 may be realized by an information recordingmedium, such as a flash memory capable of recording data and updatingrecorded data, a hard disk that is built-in or connected via a datacommunication terminal, or a memory card, and a read-write device of theinformation recording medium. Any recording device may be employedappropriately depending on the intended usage.

The memory 57 is used as a working memory of the CPU 59, and has amemory area for loading the programs executed by the CPU 59 andtemporarily storing data or the like used during execution of theprograms.

The LAN I/F unit 58 is used to perform data communication with anexternal device (for example, the server 3) and is connected to thenetwork N via a LAN to transmit and receive image data and voice data toand from the other connected devices 5 via the server 3. Any device,such as a device (wired LAN) that performs control based on 10 Base-T,100 Base-TX, or 1000 Base-T and connects to the Ethernet (registeredtrademark) or a device (wireless LAN) that performs control based on802.11a/b/g/n, may be applied as the LAN I/F unit 58 depending on aconnection mode.

The CPU 59 integrally controls the operation of the conference device 5by giving instructions or transferring data to the units of theconference device 5 based on image data input from the camera 51, voicedata input from the microphone 52, image data or voice data input fromthe other connected devices 5 via the LAN I/F unit 58, input datareceived from the operating unit 53, or programs and data recorded inthe recording device 56. For example, after the CPU 59 has established acommunication connection with the server 3 in response to a call fromthe server 3, the CPU 59 repeats a process of transmitting image datainput by the camera 51 and voice data input by the microphone 52 to theserver 3 and a process of receiving image data and voice data of theother connected devices 5 transferred by the server 3, in a parallelmanner.

Specifically, the CPU 59 performs an image processing process (to bedescribed later) on image data continually input from the camera 51during the video conference. The CPU 59 outputs the image data(processed image data) subjected to the image processing process to theLAN I/F unit 58 together with the voice data continually input from themicrophone 52, to thereby transmit the image data and the voice data tothe server 3. The CPU 59 receives image data and voice data that aretransmitted by the other connected devices 5 and that are transferred bythe server 3 via the LAN I/F unit 58, in parallel with the abovetransmission process. The CPU 59 performs a process of continuallyoutputting the received image data to the display unit 54 to display oroutput a conference screen and a process of continually outputting thereceived voice data to the speaker 55 to output audio, to therebyreproduce the images and voice input by the other connected devices 5.The image processing process need not be performed by the CPU 59 and maybe performed by a dedicated image processing circuit.

The conference device (connected device) 5 of the first embodimentperforms a predetermined image processing process on the image datainput from the camera 51, and thereafter communicates with the otherconnected device 5. The image processing process is prepared as an imagedisplay format in which an individual image processing process isdefined, and is recorded in the recording device 56 in advance. Theimage display format will be explained below. In the following, forsimplicity of explanation, it is assumed that the two connected devices5 have established communication connections with the server 3 and avideo conference is conducted between the two participating locations.

The image display format indicates a method for modifying a part or thewhole of the image data input from the camera 51 and defines the methodby an image processing process. Specifically, the image display formatserves as a rule for changing the view of an image.

FIG. 3A is a diagram illustrating an example of a field of view A1 ofthe camera 51 of the connected device 5 installed in one participatinglocation of the video conference, where three conferees P1, P2, and P3are sitting side by side in a conference room in the participatinglocation. FIG. 3B is a diagram illustrating image data I1 obtained bycapturing the field of view A1 illustrated in FIG. 3A by the camera 51.As described above, the camera 51 used in the video conference system 1includes the wide-angle lens. Therefore, if the camera 51 captures thefield of view A1, the image may be distorted due to the lenscharacteristics of the wide-angle lens. Consequently, while in actualitythe conferees P1, P2, and P3 are sitting at positions with approximatelythe same distance from the camera 51 as illustrated in FIG. 3A, theconferee P1 in the center appears smaller while the conferees P2 and P3on both sides appear greater in the generated image data as illustratedin FIG. 3B. Therefore, to conduct the video conference withoutuncomfortable feeling, it is necessary to perform a distortioncorrection process on the image data I1 generated by the camera 51 tocorrect image distortion in order to make the image closer to actualview as illustrated in FIG. 3A.

Therefore, in the first embodiment, an image display format containingall of the conferees P1, P2, and P3 (hereinafter, the image displayformat is referred to as “an overall image”) is prepared, and adistortion correction process for correcting image distortion is definedas the image processing process. FIG. 4 is a diagram illustrating anexample of overall image data 121 obtained by performing the imageprocessing process for the overall image. The image data I1 input fromthe camera 51 contains all of the conferees P1, P2, and P3 asillustrated in FIG. 3B. Therefore, if the distortion correction processis performed on the image data I1, the overall image data I21 (FIG. 4)containing the conferees in the participating location can be obtainedas processed image data in which the image distortion due to the use ofthe wide-angle lens is corrected. With the overall image data I21, theimage quality of the camera 51 using the wide-angle lens can be ensured.Therefore, conferees in the other participating location can conduct thevideo conference while viewing the conference screen that is closer tothe actual view of the conference room (FIG. 3A) and that does not causeuncomfortable feeling compared with the image data I1 that is notsubjected to the image processing process.

Meanwhile, in the video conference, in some cases, it may be desired tofocus on a specific conferee, such as a conferee who is mainly making astatement. For example, if it is desired to focus on the conferee P1 inthe center in FIG. 4 during the video conference, it may be difficult torecognize the facial expression of the conferee P1 in the overall imagedata 121 in FIG. 4 because he/she appears in a small size. Therefore, inthe first embodiment, an image display format for focusing on theconferee P1 in the center (hereinafter, the image display format isreferred to as “a partially-enlarged image”) is prepared, and a processfor extracting and enlarging an area of the conferee P1 is defined asthe image processing process. FIG. 5 is a diagram illustrating anexample of partially-enlarged image data I23 obtained by performing theimage processing process for the partially-enlarged image. With use ofthe partially-enlarged image data, it becomes possible to conduct thevideo conference while viewing a conference screen with a close-up(zoom-in) view of the conferee P1, so that it becomes possible to solvethe disadvantage that the speaker appears in a small size in the overallimage data and to conduct the video conference smoothly. The presentinvention is not limited to the partially-enlarged image that focuses onthe conferee P1 in the center. It may be possible to appropriatelyprepare an image display format that defines an image processing processfor extracting and enlarging an area of the conferee P2 on theobserver's left assuming that the conferee P2 mainly makes a statement,or an image display format that defines an image processing process forextracting and enlarging an area of the conferee P3 on the observer'sright assuming that the conferee P3 mainly makes a statement.

The flow of a process for actually exchanging the processed image databetween the connected devices 5 will be explained below. Each of theconnected devices 5 serves as a transmission-side connected device 5 torepeat the process of transmitting the processed image data (an imagedata transmission process) as described above and also serves as areception-side connected device 5 to repeat the process of receiving theprocessed image data (an image data reception process) as describedabove, in a parallel manner. FIG. 6 is a flowchart illustrating the flowof the image data transmission process, and FIG. 7 is a flowchartillustrating the flow of the image data reception process. Before astart of the image data transmission process and the image datareception process, communication connections between the server 3 andthe conference devices 5 in the respective participating locations of avideo conference are established and the video conference is started.During the video conference, voice data is also exchanged in addition tothe processed image data.

In the image data transmission process, as illustrated in FIG. 6, theCPU 59 first designates two or more types of transmission image displayformats used for transmission by the subject connected device 5 (StepS101). For example, the CPU 59 performs a process of displaying a listof transmittable image display formats on the display unit 54, andreceives designation operation to designate two or more types of imagedisplay formats via the operating unit 53. The CPU 59 designates the twoor more types of image display formats as the transmission image displayformats according to the received designation operation.

Subsequently, the CPU 59 performs a process for notifying the otherconnected device 5 of pieces of type information on the transmissionimage display formats designated at Step S101 (Step S103). As an actualprocess, the CPU 59 performs a process of notifying the server 3 of thepieces of the type information on the transmission image displayformats, and the server 3 performs a process of transferring the piecesof the type information to the other connected device 5.

The CPU 59 starts a process of capturing an image of a conferee bydriving the camera 51 and acquires image data continually input from thecamera 51 (Step S105). The CPU 59 generates the processed image data ofeach of the transmission image display formats by individuallyperforming the image processing processes defined in the two or moretypes of transmission image display formats designated at Step S101 onthe image data acquired at Step S105 (Step S107).

The CPU 59 performs a process for transmitting the processed image dataof each of the transmission image display formats to the other connecteddevice 5 (Step S109). As an actual process, the CPU 59 performs aprocess of transmitting the processed image data of each of thetransmission image display formats to the server 3, and the server 3performs a process of transferring the processed image data to the otherconnected device 5.

Thereafter, the CPU 59 determines whether the video conference hasended. If the video conference has not ended (NO at Step S111), theprocess returns to Step S105 and the above processes are repeated. Ifthe video conference has ended (YES at Step S111), the image datatransmission process is terminated.

For example, it is assumed that the conferee of the transmission-sideconnected device 5 designates two types of image display formats for theoverall image and the partially-enlarged image as the transmission imagedisplay formats. In this case, the transmission-side connected device 5notifies the connected device 5 on the receiving side (the other device)of the pieces of the type information indicating the overall image andthe partially-enlarged image via the server 3 (Step S101 to Step S103).The transmission-side connected device 5 performs, as the imageprocessing process, the distortion correction process on the image dataacquired at Step S105 to generate the overall image data 121 asillustrated in FIG. 4, and separately performs, as the image processingprocess, a process of extracting and enlarging the area of the confereeP1 in the center in FIG. 3A on the image data acquired at Step S105 togenerate the partially-enlarged image data I23 as illustrated in FIG. 5(Step S107). Then, the transmission-side connected device 5 transmitsthe overall image data and the partially-enlarged image data to theserver 3 (Step S109).

In contrast, in the image data reception process, as illustrated in FIG.7, the CPU 59 first performs a process of receiving the pieces of thetype information on the transmission image display formats that aretransmitted by the other connected device 5 and that are transferred bythe server 3 (Step S201), and performs a process of receiving theprocessed image data of each of the transmission image display formatstransferred by the server 3 (Step S203).

Subsequently, the CPU 59 sets, as initial setting of a received-imagedisplay format (hereinafter, referred to as “an initial received-imagedisplay format”), an image display format to be used for the conferencescreen by the subject connected device 5 (Step S205), and performs aprocess of outputting the processed image data of the initialreceived-image display format to the display unit 54 in order to displaythe conference screen (Step S207). The initial setting of thereceived-image display format may be performed by, for example,automatically selecting one of the pieces of the type informationreceived at Step S201 and setting the selected piece of the typeinformation as an initial value of the received-image display format, orby receiving input of operation from the conferee. In the case ofreceiving the operation, the operation can be received through the sameprocesses as those at Steps S211 and S213 described below. In this case,the CPU 59 performs a process of notifying the server 3 of the typeinformation on the initial received-image display format. At Step S207,the CPU 59 displays the conference screen by using the processed imagedata of the initial received-image display format among the pieces ofthe processed image data of the respective transmission image displayformats received at Step S203.

Subsequently, the CPU 59 determines whether the video conference hasended. If the video conference has not ended (NO at Step S209), the CPU59 determines whether operation for changing the received-image displayformat is input. If the operation for changing the received-imagedisplay format is not input (NO at Step S211), the CPU 59 causes theprocess to proceed to Step S215.

In contrast, when the operation for changing the received-image displayformat is input (YES at Step S211), the CPU 59 changes thereceived-image display format (Step S213), and the process proceeds toStep S215. The process here is performed as described below for example.Specifically, the CPU 59 performs a process of displaying, on thedisplay unit 54, a list of the two or more types of transmission imagedisplay formats designated by the other connected device 5 as imagedisplay formats available for the conference screen in accordance withthe pieces of the type information received at Step S201. Then, the CPU59 receives designation operation to designate one of the image displayformats via the operating unit 53, and sets the one designated imagedisplay format as a changed received-image display format. In this case,the CPU 59 performs a process of notifying the server 3 of the typeinformation on the changed received-image display format.

When three or more connected devices 5 are involved, it may be possibleto receive the operation for changing the received-image display formatfor each of the other connected devices 5. The same processes areperformed when the initial setting of the received-image display formatis performed by receiving input of operation. With this configuration,when a video conference is conducted among three or more participatinglocations, it becomes possible to designate the overall image for acertain connected device 5 to enable to view all of the conferees in acorresponding participating location, and designate thepartially-enlarged image for the other connected devices 5 to focus on aspecific conferee (the conferee P1 in FIG. 3A) in correspondingparticipating locations.

At Step S215, the CPU 59 performs a process of receiving the processedimage data in the received-image display format. As described above, thetransmission-side connected device 5 designates the transmission imagedisplay formats that the transmission-side connected device 5 uses fortransmission in the image data transmission process (Step S101 in FIG.6), and transmits the processed image data of each of the designatedtransmission image display formats to the server 3 (Step S109 in FIG.6). Furthermore, the reception-side connected device 5 notifies theserver 3 of the type information on the received-image display formatwhen the initial received-image display format is set at Step S205 andwhen the received-image display format is changed at Step S213. Afterreceiving the notice of the initial setting or the notice of the changeof the received-image display format, the server 3 transfers only theprocessed image data of the received-image display format among thepieces of the processed image data transmitted by the transmission-sideconnected device 5 to the reception-side connected device 5 as will bedescribed later (Step S307 in FIG. 8). At Step S215 in FIG. 7, the CPU59 receives the processed image data transferred by the server 3 asdescribed above.

The CPU 59 performs a process of outputting the processed image datareceived at Step S215 to the display unit 54 to display the conferencescreen, so that the conference screen using the processed image data ofthe changed received-image display format is displayed (Step S217).

The CPU 59 returns the process to Step S209 and repeats the aboveprocesses until the video conference ends. If the video conference hasended (YES at Step S209), the image data reception process isterminated.

For example, if the transmission-side connected device 5 designates thetwo types of the display formats, i.e., the overall image and thepartially-enlarged image, as the transmission image display formats, thereception-side connected device 5 selects one of the overall image dataand the partially-enlarged image data to display the conference screen.Specifically, if the reception-side connected device 5 sets the overallimage as the initial received-image display format, the reception-sideconnected device 5 notifies the server 3 of the type information on theoverall image and receives the overall image data transferred by theserver 3 as a reply (Steps S205 and S215). After the initial setting isperformed, the video conference is conducted while viewing theconference screen showing all of the conferees P1, P2, and P3illustrated in FIG. 3A.

Thereafter, if the conferee inputs operation for changing thereceived-image display format to the partially-enlarged image withrespect to the reception-side connected device 5 at an arbitrary timing,the reception-side connected device 5 changes the received-image displayformat to the partially-enlarged image and notifies the server 3 of thetype information on the partially-enlarged image (if YES at Step S211,then the process proceeds to S213). The reception-side connected device5 receives the partially-enlarged image data transferred by the server 3as a reply (Step S215). After the change operation is performed, thevideo conference is conducted while viewing the conference screenshowing a close-up view of the conferee P1 in the center in FIG. 3A.

FIG. 8 is a diagram illustrating the flow of data transmitted andreceived between the server 3 and the connected devices (conferencedevices) 5. FIG. 9 is a diagram illustrating the flow of datatransmitted and received between the server 3 and the connected devices(conference devices) 5 when the operation for changing thereceived-image display format is input to the reception-side connecteddevice 5. The processed image data is exchanged between the connecteddevices 5 by causing the server 3 to intervene between the connecteddevices 5 that repeat the image data transmission process and the imagedata reception process in a parallel manner. FIG. 8 and FIG. 9illustrate the data flow in the case where the processed image data istransmitted from the transmission-side connected device 5 to thereception-side connected device 5 on the assumption that one of theconnected devices 5 serves as a transmission side and the other one ofthe connected devices 5 serves as a reception side. However, inactuality, the reception-side connected device 5 also serves as thetransmission side and the transmission-side connected device 5 alsoserves as the reception side, so that the same processes are performedin parallel in the reverse direction.

As illustrated in FIG. 8, the transmission-side connected device 5notifies the server 3 of the pieces of the type information on thetransmission image display formats and starts transmission of the piecesof the processed image data (Step S301). The process at this stepcorresponds to the processes from Steps S103 to S109 in FIG. 6. Inresponse to the above process, the server 3 transfers the pieces of thetype information on the transmission image display formats to thereception-side connected device 5 and starts transferring the processedimage data of each of the transmission image display formats to thereception-side connected device 5 (Step S303).

In contrast, when receiving the pieces of the type information on thetransmission image display formats transferred by the server 3, thereception-side connected device 5 sets the initial received-imagedisplay format and notifies the server 3 of the type information on theinitial received-image display format (Step S305). The process at thisstep corresponds to the process at Step S205 in FIG. 7. After receivingthe notice of the initial setting of the received-image display formatfrom the reception-side connected device 5 as described above, theserver 3 transfers, to the reception-side connected device 5, onlyprocessed image data of the received-image display format among thepieces of the processed image data continually transmitted by thetransmission-side connected device 5 (Step S307). In actuality, toenable the process, the server 3 manages a list of the transmissionimage display formats designated by each of the connected devices 5 anda list of the received-image display formats designated by each of theconnected devices 5. The server 3 refers to the lists to perform aprocess of selectively transferring, to the reception-side connecteddevice 5, the processed image data of each of the two or more types oftransmission image display formats transmitted by the transmission-sideconnected device 5.

Furthermore, as illustrated in FIG. 9, when the operation for changingthe received-image display format is performed, the reception-sideconnected device 5 notifies the server 3 of the type information on thechanged received-image display format (Step S401). The process at thisstep corresponds to the process at Step S213 in FIG. 7. When receivingthe notice of the change of the received-image display format from thereception-side connected device 5 as described above, the server 3switches to the changed received-image display format and transfers theprocessed image data of the changed received-image display format to thereception-side connected device 5 (Step S403). In this case, the server3 updates the list of the received-image display formats as describedabove.

As described above, according to the first embodiment, the conferencedevice (connected device) 5 can prepare the image display formats, ineach of which the image processing process is individually defined inadvance, and can designate, as the transmission image display formats,two or more types of image display formats that the connected device 5uses for transmission. When transmitting the image data to the otherconnected device 5, the connected device 5 can individually perform theimage processing process defined in the transmission image displayformats to generate the processed image data of each of the transmissionimage display formats. The processed image data of each of the imagedisplay formats generated as above can be transmitted to the otherconnected device 5 via the server 3. In contrast, when receiving theimage data from the transmission-side connected device 5, the conferencedevice (connected device) 5 can change the received-image display formatby setting, as the initial received-image display format, one of theimage display formats that are designated as the transmission imagedisplay formats by the other connected device 5, or by receiving inputof operation from the conferee. Then, the conference screen using theprocessed image data of the received-image display format can bedisplayed. Therefore, it is possible to display the conference screen asdesired by the conferee.

In particular, in the first embodiment, it is possible to acquire imagedata containing all of the conferees at any time by using the camera 51including the wide-angle lens. It is also possible to, for example,perform the distortion correction process on the acquired image data togenerate the overall image data according to the image display format(the overall image) that defines the image processing process forcorrecting image distortion, generate the partially-enlarged image dataaccording to the image display format (the partially-enlarged image)that defines the image processing process of extracting and enlarging anarea of a specific conferee (the conferee P1 in the center in FIG. 3A)on the image data, and transmit the pieces of the image data to theother connected device 5 via the server 3. The reception-side connecteddevice 5 can select and use one of the overall image data and thepartially-enlarged image data to display the conference screen.

Furthermore, the server 3 can receive the processed image data of eachof the transmission image display formats from the transmission-sideconnected device 5 and transfer the processed image data to thereception-side connected device 5. Because the server 3 receives theprocessed image data of each of the transmission image display formatsfrom the transmission-side connected device 5 as described above, afterreceiving the notice of the initial setting or the notice of the changeof the received-image display format from the reception-side connecteddevice 5, the server 3 can transfer only the processed image data of thedesignated received-image display format to the reception-side connecteddevice 5. Therefore, it becomes possible to transfer only the processedimage data to be displayed on the conference screen by each of theconnected devices 5 to each of the connected devices 5, so that thenetwork bandwidth used for data communication between the server 3 andthe connected devices 5 can be reduced. Therefore, even when a line withlower network bandwidth capability is used, it is possible to stablyperform data communication.

In the first embodiment, a case is explained that the two connecteddevices 5 are used. However, the same can apply when three or moreconnected devices 5 are connected to the server 3 to conduct a videoconference among the three or more participating locations.Specifically, the server 3 performs a process of selectivelytransferring the processed image data of each of the transmission imagedisplay formats transmitted by the transmission-side connected device 5to each of the reception-side connected devices 5 by referring to a listof the transmission image display formats or the received-image displayformats that are managed and updated as described above for each of theconnected devices 5.

First Modification

In the first embodiment described above, it is explained that thetransmission-side connected device 5 transmits the processed image dataof each of the transmission image display formats to the server 3, andthe server 3 transfers only the processed image data of thereceived-image display format designated by the reception-side connecteddevice 5 to the reception-side connected device 5. In contrast, thetransmission-side connected device 5 may transmit only the processedimage data of the received-image display format designated by thereception-side connected device 5.

FIG. 10 is a diagram illustrating the flow of data transmitted andreceived between the server 3 and the connected devices (conferencedevices) 5 according to a first modification. FIG. 11 is a diagramillustrating the flow of data transmitted and received between theserver 3 and the connected devices (conference devices) 5 when theoperation for changing the received-image display format is input to thereception-side connected device 5. In FIG. 10 and FIG. 11, the sameprocesses as those described in the first embodiment are denoted by thesame reference numerals and symbols, and explanation thereof will beomitted.

As illustrated in FIG. 10, when receiving the notice of the initialsetting of the received-image display format from the reception-sideconnected device 5, the server 3 transfers only the processed image dataof the received-image display format to the reception-side connecteddevice 5 similarly to the first embodiment as described above (StepS307). However, in the first modification, the received notice of theinitial setting (the type information on the received-image displayformat) is subsequently transferred to the transmission-side connecteddevice 5 (Step S509). In response to the above process, thetransmission-side connected device 5 transmits only the processed imagedata of the notified received-image display format to the server 3 (StepS511). In this case, the transmission-side connected device 5 only needsto generate the processed image data of the received-image displayformat. The server 3 transfers the processed image data transmitted bythe transmission-side connected device 5 to the reception-side connecteddevice 5 (Step S513).

Furthermore, as illustrated in FIG. 11, when receiving the notice of thechange of the received-image display format from the reception-sideconnected device 5, the server 3 transfers the received notice of thechange (the type information on the received-image display format) tothe transmission-side connected device 5 (Step S605). In response to theabove process, the transmission-side connected device 5 transmits onlythe processed image data of the notified received-image display formatto the server 3 (Step S607). In this case, similarly to Step S511 inFIG. 10, the transmission-side connected device 5 only needs to generatethe processed image data of the received-image display format. Theserver 3 transfers the processed image data transmitted by thetransmission-side connected device 5 to the reception-side connecteddevice 5 (Step S609).

In the first modification, after the reception-side connected device 5sets the initial received-image display format, the transmission-sideconnected device 5 transmits only the processed image data of thereceived-image display format designated by the reception-side connecteddevice 5 (Step S511 in FIG. 10 and Step S607 in FIG. 11). Therefore,when the received-image display format is changed, the transmission-sideconnected device 5 needs to change the image display format of theprocessed image data to be transmitted.

Furthermore, when three or more connected devices 5 have communicationconnections to the server 3 to conduct a video conference, in somecases, the transmission-side connected device 5 may need to add theprocessed image data to be transmitted. For example, when the fourconnected devices 5-1 to 5-4 illustrated in FIG. 1 have communicationconnections to the server 3 to conduct a video conference, and if theconnected devices 5-1 to 5-3 designate the overall image with respect tothe connected device 5-4, the connected device 5-4 generates only theoverall image data and transmits the overall image to the server 3. Inthis state, if the connected device 5-1 changes the received-imagedisplay format with respect to the connected device 5-4 to thepartially-enlarged image, the connected device 5-4 needs to newlygenerate the partially-enlarged image data in addition to the overallimage data and transmit the partially-enlarged image data to the server3. In contrast, when the connected devices 5-1 and 5-2 designate theoverall image with respect to the connected device 5-4 and the connecteddevice 5-3 designates the partially-enlarged image with respect to theconnected device 5-4, the connected device 5-4 generates the overallimage data and the partially-enlarged image data and transmits thepieces of the image data to the server 3. In this state, if theconnected device 5-1 changes the received-image display format withrespect to the connected device 5-4 to the partially-enlarged image, theconnected device 5-4 does not need to generate new processed image data.This is because, because the connected device 5-4 has already generatedthe partially-enlarged image data for the connected device 5-3 andtransmitted the partially-enlarged image data to the server 3, it issufficient to cause the server 3 to transfer the partially-enlargedimage data to the connected device 5-1.

FIG. 12 is a flowchart illustrating the flow of a received-image displayformat switching process performed by the server 3 when the server 3receives the notice of the change of the received-image display formatfrom the reception-side connected device 5 in the first modification.

In the received-image display format switching process, as illustratedin FIG. 12, the server 3 receives, from the reception-side connecteddevice 5, the notice of the change of the received-image display formatdesignated for the transmission-side connected device 5 (Step S701).When receiving the notice of the change, the server 3 determines whetherthe changed received-image display format is designated by the otherconnected devices 5. Similarly to the first embodiment as describedabove, the server 3 manages the lists of the transmission image displayformats and the received-image display formats of each of the connecteddevices 5, and performs the determination by referring to the lists. Ifthe changed received-image display format is designated by the otherconnected devices 5 (YES at Step S703), the server 3 transfers theprocessed image data of the changed received-image display format to thereception-side connected device 5 (Step S705).

In contrast, if the changed received-image display format is notdesignated by the other connected devices 5 (NO at Step S703), theserver 3 transfers the notice of the change of the received-imagedisplay format to the transmission-side connected device 5 to requesttransmission of the processed image data (Step S707). In response to theabove process, the transmission-side connected device 5 generates newprocessed image data of the changed received-image display format andtransmits the generated processed image data to the server 3. The server3 transfers the processed image data of the changed received-imagedisplay format transmitted by the transmission-side connected device 5as described above to the reception-side connected device 5 (Step S709).

In the first modification, the CPU 59 of the connected device 5determines whether the notice of the initial setting and the notice ofthe change are transferred by the server 3 after Step S105 in FIG. 6,and, after the notices are transferred, the CPU 59 performs a process ofgenerating only the processed image data of the received-image displayformat designated by the other connected device 5 and transmitting thegenerated processed image data to the server 3 instead of the processesat Step S107 and S109. Furthermore, the CPU 59 of the connected device 5performs the image data reception process in the same manner as in FIG.7.

As described above, according to the first modification, it is possibleto achieve the same advantageous effects as those of the firstembodiment. Furthermore, after the initial setting of the received-imagedisplay format is performed, each of the connected devices 5 cangenerate only the processed image data of the received-image displayformat designated by the other connected devices 5 from among thetransmission image display formats designated by each of the connecteddevices 5, and transmit the generated processed image data to the server3. Therefore, it becomes possible to reduce the usage rate of the CPU 59of each of the connected devices 5. If the image processing process isperformed by a dedicated image processing circuit, it is possible toreduce the usage rate of the image processing circuit. As a result, itis possible to reduce power consumption of each of the connected devices5.

Second Modification

In the first embodiment described above, the image processing process isperformed by the connected device 5. However, the image processingprocess need not be performed by the connected device 5 and may beperformed by the server 3.

FIG. 13 is a flowchart illustrating the flow of an image data transferprocess performed by the server 3 according to a second modification.Before a start of the image data transmission process and the image datareception process, communication connections between the server 3 andthe conference devices 5 in the respective participating locations of avideo conference are established and the video conference is started. Inthe second modification, the image display format that defines the imageprocessing process is prepared on the server 3 side.

In the image data transfer process, as illustrated in FIG. 13, theserver 3 first receives pieces of the type information on two or moretypes of transmission image display formats transmitted by thereception-side connected device 5 (Step S801), and receives image datatransmitted by the transmission-side connected device 5 (Step S803).

The server 3 individually performs the image processing process definedin the two or more types of transmission image display formats receivedat Step S801 on the image data received at Step S803 to thereby generatethe processed image data of each of the transmission image displayformats (Step S805).

The server 3 determines whether the type information on thereceived-image display format is received from the reception-sideconnected device 5, that is, whether a notice of initial setting or anotice of a change of the received-image display format are received.While the type information on the received-image display format is notreceived from the reception-side connected device 5 (NO at Step S807),the server 3 transmits the processed image data of each of thetransmission image display formats generated at Step S805 to thereception-side connected device 5 (Step S809). In contrast, whenreceiving the type information on the received-image display format fromthe reception-side connected device 5 (YES at Step S807), the server 3transmits only the processed image data of the received-image displayformat among the pieces of the processed image data of the respectivetransmission image display formats generated at Step S805 to thereception-side connected device 5 (Step S811). Meanwhile, afterreceiving the type information on the received-image display format fromthe other connected devices 5, each of the connected devices 5 maygenerate only the processed image data of the received-image displayformat designated by the other connected devices 5.

Thereafter, the server 3 determines whether the video conference hasended. If the video conference has not ended (NO at Step S813), theprocess returns to Step S803 and the above processes are repeated. Ifthe video conference has ended (YES at Step S813), the image datatransfer process is terminated.

In the second modification, the CPU 59 of the connected device 5 doesnot perform the process at Step S107 in the image data transmissionprocess in FIG. 6, and performs a process of transmitting the image dataacquired at Step S105 to the server 3 instead of the process at StepS109. Furthermore, the CPU 59 of the connected device 5 performs theimage data reception process in the same manner as in FIG. 7.

As described above, according to the second modification, it is possibleto achieve the same advantageous effects as those of the firstembodiment. Furthermore, after the initial setting of the received-imagedisplay format is performed, the server 3 can generate only necessaryprocessed image data according to the received-image display formatdesignated by each of the connected devices 5, and transmit theprocessed image data to each of the connected devices 5.

Third Modification

In the first embodiment, the image processing is performed on all of theimage display formats immediately after the conference is started. In athird modification, an example will be explained in which the imageprocessing is performed on a single image display format immediatelyafter the conference is started. FIG. 14 is a flowchart illustrating theflow of an image data transmission process according to the thirdmodification. FIG. 14 illustrates the flow of the image processingperformed on the image display format by the transmission-side connecteddevice 5 performs after a request for an image display format desired tobe displayed is received from the reception-side connected device 5.

The CPU 59 of the transmission-side connected device 5 notifies theother connected device 5 of the type information on the image displayformat to be transmitted (the transmission image display format) (StepS901). The CPU 59 performs image processing according to a default imagedisplay format and transmits the processed image data to the network(Step S902). The default image display format is an arbitrary one ofimage display formats that can be transmitted by the transmission-sideconnected device 5. For example, information on which format serves asthe default image display format is recorded in advance in the recordingdevice of the transmission-side connected device 5.

The CPU 59 determines whether the reception-side connected device 5 hasrequested a change of the image display format to be displayed (StepS903). When the change has been requested (YES at Step S903), the CPU 59determines whether the image processing according to the requested imagedisplay format is being performed (Step S904).

If the image processing according to the requested image display formatis not being performed (NO at Step S904), the CPU 59 performs the imageprocessing according to the newly-requested image display format inaddition to the image processing according to the default image displayformat, and transmits the processed image data to the network (StepS905).

Thereafter (NO at Step S903, YES at Step S904, or after completion ofStep S905), the CPU 59 determines whether the video conference has ended(Step S906). If the video conference has not ended (NO at Step S906),the process returns to Step S903 and the above processes are repeated.If the video conference has ended (YES at Step S906), the image datatransmission process is terminated.

In the example in FIG. 14, the image processing is added upon receptionof the request for the change of the image display format from thereception-side connected device 5. In contrast, when there is an imagedisplay format that is not displayed by any of the reception-sideconnected devices 5, the transmission-side connected device 5 may cancelthe image processing according to the image display format and cancelthe transmission via the network.

Through the processes as illustrated in FIG. 14, the transmission-sideconnected device 5 need not transmit multiple image display formats tothe network immediately after the conference is started. Furthermore,the transmission-side connected device 5 need not perform the imageprocessing according to the image display format that is not displayedby the reception-side connected device 5.

FIG. 15 is a diagram illustrating the flow of data transmitted andreceived between the server and the conference devices according to thethird modification.

When the conference is started, the transmission-side connected device 5notifies the server 3 of the type information on the transmission imagedisplay format and starts transmitting an image of the default imagedisplay format (Step S1001). The server 3 transfers the notice of thetype information on the image display format and the processed imagedata received from the transmission-side connected device 5 to each ofthe reception-side connected devices 5 (Step S1002).

When switching to another image display format that differs from thecurrent image display format, the reception-side connected device 5notifies the server 3 of an image display format desired to be displayed(received-image display format) (Step S1003). The server 3 notifies thetransmission-side connected device 5 of the image display formatrequested by the reception-side connected device 5 (Step S1004).

When receiving the notice, the transmission-side connected device 5transmits the requested image display format to the server 3 (StepS1005). The server 3 transmits only an image of the image display formatrequested by the reception-side connected device 5 to the reception-sideconnected device 5 (Step S1006).

In some cases, the transmission-side connected device 5 has alreadytransmitted a corresponding image display format to the server 3 whenthe reception-side connected device sends a request for a change of theimage display format to be displayed. In this case, the server 3 maytransmit the corresponding image display format that has been alreadyreceived to the reception-side connected device 5 that has sent therequest, without notifying the transmission-side connected device 5 ofthe change request. For example, this may occur when a conference isconducted among multiple locations.

Fourth Modification

In general, the load of the CPU 59 of the connected device 5 greatlyincreases due to image processing, so that it is difficult tosimultaneously perform a plurality of types of image processing by asingle connected device 5. Therefore, it may be possible to set, inadvance before shipment, the number (a predetermined number) of types ofimage processing that can simultaneously be performed by a singleconnected device 5, and if it is necessary to perform a greater numberof types of image processing than the predetermined number, the server 3may be used. With this configuration, it becomes possible to perform aplurality of types of image processing.

FIG. 16 is a flowchart illustrating the flow of an image datatransmission process according to a fourth modification configured asdescribed above.

The processes from Step S1101 to Step S1104 are the same as those fromStep S901 to Step S904 in FIG. 14; therefore, the same explanation willnot be repeated.

At Step S1104, if the image processing according to the requested imagedisplay format is not performed (NO at Step S1104), the CPU 59determines whether the number obtained by adding one that corresponds tothe number of a new image display format to the number of the imagedisplay formats that are currently subjected to the image processing bythe transmission-side connected device 5 is equal to or smaller than thepredetermined number (Step S1105). If the number is equal to or smallerthan the predetermined number (YES at Step S1105), the CPU 59 performsthe image processing according to the requested image display format inaddition to the current image processing, and transmits the processedimage data to the network (Step S1108). It is assumed that thepredetermined number is stored in, for example, the storage device ofthe transmission-side connected device 5 in advance.

If the number is greater than the predetermined number (NO at StepS1105), the CPU 59 requests the server 3 to perform the image processingaccording to the requested image display format (Step S1106). The CPU 59transmits an original image that is to be subjected to the imageprocessing by the server 3, that is, an image that has not beensubjected to the image processing, to the server 3 (Step S1107).

Thereafter (NO at Step S1103, YES at Step S1104, after completion ofStep S1107, or after completion of Step S1108), the CPU 59 determineswhether the video conference has ended (Step S1109). If the videoconference has not ended (NO at Step S1109), the process returns to StepS1103 and repeats the above processes. If the video conference has ended(YES at Step S1109), the image data transmission process is terminated.

FIG. 17 is a flowchart illustrating the flow of the image processingperformed by the server 3 according to the fourth modification.

When the conference is started, the server 3 determines whether thetransmission-side connected device 5 has requested the image processing(Step S1201). If the image processing has been requested (YES at StepS1201), the server 3 performs the image processing by using an originalimage received from the transmission-side connected device 5 andtransmits the processed image to the reception-side connected device 5(Step S1202).

Thereafter (NO at Step S1201 or after completion of Step S1202), theserver 3 determines whether the video conference has ended (Step S1203).If the video conference has not ended (NO at Step S1203), the processreturns to Step S1201 and the above processes are repeated. If the videoconference has ended (YES at Step S1203), the image processing isterminated.

FIG. 18 is a diagram illustrating the flow of data transmitted andreceived between the server and the conference devices according to thefourth modification.

When the conference is started, the transmission-side connected device 5notifies the server 3 of the type information on the transmission imagedisplay format and starts transmitting an image of the default imagedisplay format (Step S1301). The server 3 notifies the reception-sideconnected device 5 of the type information on the transmission imagedisplay format received from the transmission-side connected device 5and transmits the processed image data to the reception-side connecteddevice 5 (Step S1302).

When changing the desired image display format to be displayed, thereception-side connected device 5 notifies the server 3 of the imagedisplay format (received-image display format) (Step S1303). The server3 transmits the request for the change of the image display formatreceived from the reception-side connected device 5 to thetransmission-side connected device 5 (Step S1304).

If the transmission-side connected device 5 is not able to perform theimage processing according to the requested image display format, thetransmission-side connected device 5 notifies the server 3 of therequest for the image processing (Step S1305). Subsequently, thetransmission-side connected device 5 transmits an image that has notbeen subjected to the image processing to the server 3 (Step S1306).

The server 3 performs the image processing according to the imagedisplay format requested by the reception-side connected device 5 atStep S1303 on the received image, and transmits the image subjected tothe image processing to the reception-side connected device 5 (StepS1307).

Second Embodiment

A configuration of a display control system according to a secondembodiment is the same as the configuration illustrated in FIG. 1;therefore, explanation thereof will be omitted. Furthermore, theinternal configuration of a conference device according to the secondembodiment is the same as the configuration illustrated in FIG. 2B;therefore, explanation thereof will be omitted.

FIG. 19 is a block diagram illustrating a functional configuration of aconference device 5 b according to the second embodiment. As illustratedin FIG. 19, the CPU 59 of the conference device 5 b loads a programstored in the recording device 56 or the like into the memory 57 andexecutes the program to thereby function as an image processing unit2401, a receiving unit 2402, a display control unit 2403, and atransmitting unit 2404.

In the explanation below, a case is described that a conference device 5b-1 (subject device) installed in a first location conducts a videoconference with a conference device 5 b-2 (external device) installed ina second location (another location). However, the same applies when thevideo conference is conducted with conference devices 5 b-3 and 5 b-4installed in the other locations. Furthermore, in the explanation below,a case is described that the conference device 5 b-1 functions as bothof a captured-image transmission side and a captured-image receptionside. However, it is sufficient that the conference device 5 b-1functions as at least the image processing unit 2401 and the receivingunit 2402 when serving as the captured-image transmission side (anexample of an image processing apparatus), and functions as at least thedisplay control unit 2403 and the transmitting unit 2404 when serving asthe captured-image reception side.

The image processing unit 2401 performs predetermined image processingon a captured image obtained by capturing, by the camera 51, the firstlocation in which the conference device 5 b-1 is installed (or imageprocessing designated by the conference device 5 b-2 installed in thesecond location), and transmits the captured image subjected to theimage processing to the conference device 5 b-2 (external device)installed in the second location (another location) different from thefirst location. When performing the image processing designated by theconference device 5 b-2, the image processing unit 2401 performs theimage processing on the captured image according to processinginformation received by the receiving unit 2402 to be described below.

The receiving unit 2402 receives the processing information (forexample, an image processing method or the like) that designates theimage processing to be performed on the captured image from theconference device 5 b-2 installed in the second location.

The display control unit 2403 acquires the captured image subjected tothe image processing from the conference device 5 b-2 (external device),which is installed in the second location (another location) differentfrom the first location where the subject device is installed and whichperforms the image processing on the captured image of the secondlocation, and displays the captured image on the display unit 54 of theconference device 5 b-1 installed in the first location.

The transmitting unit 2404 transmits the processing information (forexample, an image processing method or the like) that designates theimage processing to be performed on the captured image to the conferencedevice 5 b-2 in the second location, and causes the conference device 5b-2 in the second location to perform the image processing according tothe processing information.

The flow of video conference control performed by the conference device5 b-1 serving as the captured-image transmission side will be explainedbelow with reference to FIG. 20. FIG. 20 is a flowchart illustrating theflow of the video conference control performed by the informationprocessing apparatus serving as the captured-image transmission side.

The receiving unit 2402 receives the processing information from theconference device 5 b-2 in the second location serving as thecaptured-image reception side (Step S1401). In the second embodiment,the receiving unit 2402 receives processing information for performingimage processing to enlarge a part of the captured image (for example, acenter portion of the captured image) (hereinafter, referred to as“partial enlargement”), and processing information for performing imageprocessing to correct distortion of the captured image (hereinafter,referred to as “distortion correction”).

In the second embodiment, the receiving unit 2402 receives theprocessing information for performing the image processing of thepartial enlargement or the distortion correction as an image processingmethod to be performed by the subject device. Any image processing maybe designated by the processing information and may be executed as longas the subject device can execute the image processing. For example, thereceiving unit 2402 may receive, from the conference device 5 b-2installed in the second location, processing information for performingimage processing, such as extraction of a partial image from the imagecaptured by the camera 51 of the conference device 5 b-1 installed inthe first location, brightness adjustment to optimize the contrast ofeach of pixels of the captured image (a so-called bright face function),or backlight adjustment to correct the captured image that is dark dueto the backlight, and the image processing unit 2401 (to be describedlater) may perform the image processing, such as the extraction of apartial image from the captured image, the bright face function, or thebacklight adjustment. The conference device 5 b on the captured-imagetransmission side may transmit the image processing that thetransmission-side conference device 5 b can perform (for example, thepartial enlargement, the distortion correction, or the like) to theconference device 5 b on the captured-image reception side in advance,and may receive the processing information designating the imageprocessing that the transmission-side conference device 5 b can perform.Alternatively, the conference device 5 b on the captured-imagetransmission side may receive the processing information designatingimage processing desired by a user of the conference device 5 b on thecaptured-image reception side regardless of whether thetransmission-side conference device 5 b is able to perform the imageprocessing, and may perform predetermined image processing when thetransmission-side conference device 5 b is unable to perform the imageprocessing according to the received processing information.

The image processing unit 2401 acquires, from the camera 51, the imagecaptured by the camera 51 (Step S1402). Subsequently, the imageprocessing unit 2401 performs image processing on the acquired imageaccording to the processing information received by the receiving unit2402 (Step S1403). For example, if the received processing informationdesignates the partial enlargement as an image processing method of theimage processing, the image processing unit 2401 obtains the image dataI23 by performing, on the acquired image data I1, the partialenlargement to enlarge the conferee P1 who is speaking among the threeconferees P1, P2, and P3 contained in the image data I1 as illustratedin FIG. 5.

In contrast, if the received processing information designates thedistortion correction, the image processing unit 2401 obtains the imagedata 121 by performing, on the acquired image data I1, the distortioncorrection to reduce the size of the peripheral portion (areascontaining the conferees P2 and P3) of the image data I1 as illustratedin FIG. 4.

The image processing unit 2401 transmits the captured image subjected tothe image processing to the conference device 5 b-2 (another informationprocessing apparatus) installed in the second location via the network N(Step S1404). Therefore, the conference device 5 b-2 installed in thesecond location can display the captured image subjected to the desiredimage processing on the display unit 54 of the conference device 5 b-2installed in the second location. Consequently, when the captured imageof the first location is subjected to the image processing andthereafter transmitted to the conference device 5 b-2 installed in thesecond location, it is possible to match the captured image that aconferee in the second location wants to view and the image to beactually displayed, so that the user's usability can be improved.

Furthermore, the conference device 5 b-2 installed in the secondlocation serving as the captured-image reception side need not performimage processing to obtain a captured image as desired by the user ofthe subject device. Therefore, it becomes not necessary to store thecharacteristics of the camera 51 (for example, the type of a lens of thecamera 51) of the conference device 5 b-1 installed in the firstlocation serving as the captured-image transmission side. Moreover, theconference device 5 b-2 installed in the second location serving as thecaptured-image reception side need not perform image processing on thecaptured image to be displayed. Therefore, it becomes possible to reduceprocessing load on the captured image due to the image processing,enabling to improve the processing performance of other processes.

The image processing unit 2401 and the receiving unit 2402 repeat theprocesses from Step S1402 to Step S1404 until the video conference withthe conference device 5 b-2 installed in the second location ends (NO atStep S1405).

The flow of video conference control performed by the conference device5 b-1 serving as the captured-image reception side will be explainedbelow with reference to FIG. 21. FIG. 21 is a flowchart illustrating theflow of the video conference control performed by the informationprocessing apparatus serving as the captured-image reception side.

The transmitting unit 2404 determines whether to transmit the processinginformation to the conference device 5 b-2 on the captured-imagetransmission side according to operation input through the operatingunit 53 by a user (for example, a conferee) of the conference device 5b-1 installed in the first location (Step S1501).

When transmitting the processing information to the conference device 5b-2 on the captured-image transmission side (YES at Step S1501), thetransmitting unit 2404 transmits the processing information designatingthe image processing method input through the operating unit 53 to theconference device 5 b-2 installed in the second location via the networkN (Step S1502). In the second embodiment, the transmitting unit 2404transmits the processing information for performing the image processingto perform the partial enlargement or the distortion correction to theconference device 5 b-2 installed in the second location. Furthermore,in the second embodiment, a general-purpose Internet protocol is usedwhen the transmitting unit 2404 transmits the processing information tothe conference device 5 b-2 installed in the second location.

Therefore, it is possible to cause the conference device 5 b-2 installedin the second location to perform the image processing on the capturedimage of the second location as desired by the user (for example, theconferee) of the conference device 5 b-1 installed in the firstlocation. Consequently, when the captured image of the second locationis subjected to the image processing and thereafter transmitted to theconference device 5 b-1 installed in the first location, it is possibleto match the captured image that a conferee in the first location wantsto view and the image to be actually displayed, so that the user'susability can be improved.

Furthermore, the conference device 5 b-1 installed in the first locationserving as the captured-image reception side need not perform imageprocessing to obtain a captured image as desired by the user of thesubject device. Therefore, it becomes not necessary to store thecharacteristics of the camera 51 (for example, the type of a lens of thecamera 51) of the conference device 5 b-2 installed in the secondlocation serving as the captured-image transmission side. Moreover, theconference device 5 b-1 installed in the first location serving as thecaptured-image reception side need not perform image processing on thecaptured image to be displayed. Therefore, it becomes possible to reduceprocessing load on the captured image due to the image processing,enabling to improve the processing performance of other processes.

In the second embodiment, the transmitting unit 2404 transmits theprocessing information for performing the image processing of thepartial enlargement or the distortion correction. Any image processingmay be designated by the processing information and may be performed aslong as the conference device 5 b-2 installed in the second location canperform the image processing. For example, the transmitting unit 2404may transmit, to the conference device 5 b-2 installed in the secondlocation, processing information for performing image processing, suchas extraction of a partial image from the image captured by the camera51 of the conference device 5 b-2 installed in the second location,brightness adjustment to optimize the contrast of each of pixels of thecaptured image (a so-called bright face function), or backlightadjustment to correct the captured image that is dark due to thebacklight, in order to perform the image processing, such as theextraction of a partial image from the captured image, the bright facefunction, or the backlight adjustment. The conference device 5 b on thecaptured-image reception side may acquire the image processing that thetransmission-side conference device 5 b can perform (for example, thepartial enlargement, the distortion correction, or the like) from theconference device 5 b on the captured-image transmission side inadvance, and may transmit the processing information designating theimage processing that the conference device 5 b on the captured-imagetransmission side can perform. Alternatively, the conference device 5 bon the captured-image reception side may transmit the processinginformation designating image processing desired by a user regardless ofwhether the transmission-side conference device 5 b is able to performthe image processing, and may receive the captured image subjected tothe predetermined image processing when the transmission-side conferencedevice 5 b is unable to the image processing according to thetransmitted processing information.

When the processing information is transmitted (Step S1502 or Step S1506to be described later) or when transmission of the processinginformation is not instructed (NO at Step S1501), the display controlunit 2403 receives the captured image subjected to the image processingdesignated by the transmitted processing information from the conferencedevice 5 b-2 on the captured-image transmission side (the conferencedevice 5 b-2 installed in the second location) via the network N (StepS1503). Then, the display control unit 2403 displays the receivedcaptured image on the display unit 54.

Subsequently, the transmitting unit 2404 determines whether the videoconference has ended (Step S1504). If the video conference has not ended(NO at Step S1504), the transmitting unit 2404 determines whether aninstruction to change the image processing performed by the conferencedevice 5 b-2 installed in the second location is input via the operatingunit 53 (Step S1505). If the instruction to change the image processingis not input (NO at Step S1505), the display control unit 2403repeatedly receives and displays the captured image until the videoconference ends.

In contrast, if the instruction to change the image processing is input(YES at Step S1505), the transmitting unit 2404 re-transmits theprocessing information input via the operating unit 53 to the conferencedevice 5 b-2 installed in the second location via the network N (StepS1506). For example, if the distortion correction is designated as theimage processing by the processing information transmitted at StepS1502, the transmitting unit 2404 transmits the processing informationfor performing the image processing to perform the partial enlargementto the conference device 5 b-2 installed in the second location.

The flow of data (captured image) transmitted and received between theconference devices 5 b-1 and 5 b-2 installed in the participantlocations of a video conference and the server 3 will be explained belowwith reference to FIG. 22. FIG. 22 is a sequence diagram illustratingthe flow of data in a display control system according to the secondembodiment. In the explanation below, a case is described that thecaptured image flows unidirectionally from the conference device 5 b-1installed in the first location to the conference device 5 b-2 installedin the second location on the assumption that the conference device 5b-1 installed in the first location serves as the captured-imagetransmission side and the conference device 5 b-2 installed in thesecond location serves as the captured-image reception side. However,when the video conference is actually performed, the captured imageflows bi-directionally between the conference device 5 b-1 installed inthe first location and the conference device 5 b-2 installed in thesecond location.

When a user of the conference device 5 b-2 in the second location inputsthe processing information indicating the image processing method or thelike via the operating unit 53, the conference device 5 b-2 (thetransmitting unit 2404) in the second location transmits the processinginformation designating the input image processing method to the server3 (Step S1601). The server 3 transfers the processing informationtransmitted by the conference device 5 b-2 in the second location to theconference device 5 b-1 in the first location (Step S1602).

When receiving the processing information from the server 3, theconference device 5 b-1 (the image processing unit 2401) in the firstlocation performs the image processing according to the image processingmethod designated by the received processing information (an imageprocessing method desired by a conferee participating in a videoconference in the second location) on the image captured by the camera51. Then, the conference device 5 b-1 in the first location transmitsthe captured image subjected to the image processing according to thedesired image processing method (for example, the image data 121illustrated in FIG. 4 or the image data I23 illustrated in FIG. 5) tothe server 3 (Step S1603). The server 3 transfers the captured imagetransmitted by the conference device 5 b-1 in the first location to theconference device 5 b-2 in the second location (Step S1604).

In FIG. 22, an example is illustrated in which the captured image istransmitted and received between the two conference devices 5 b.However, the same applies when the captured image is transmitted andreceived between more than two conference devices 5 b.

With reference to FIG. 23, explanation will be given below of the flowof data (captured image) transmitted and received between the conferencedevices 5 b-1 and 5 b-2 and the server 3 when an instruction to changethe image processing method is input during the video conference. FIG.23 is a sequence diagram illustrating the flow of data in the displaycontrol system according to the second embodiment.

When an instruction to change the image processing method is inputduring the video conference, the conference device 5 b-2 in the secondlocation (the transmitting unit 2404) transmits the processinginformation designating the changed image processing method to theserver 3 (Step S1701). The server 3 transfers the processing informationtransmitted by the conference device 5 b-2 in the second location to theconference device 5 b-1 in the first location (Step S1702).

When receiving the processing information from the server 3, theconference device 5 b-1 (the image processing unit 2401) in the firstlocation performs the image processing according to the changed imageprocessing method designated by the received processing information onthe image captured by the camera 51. Then, the conference device 5 b-1in the first location transmits the captured image subjected to theimage processing according to the changed image processing method (forexample, the image data 121 illustrated in FIG. 4 or the image data I23illustrated in FIG. 5) to the server 3 (Step S1703). The server 3transfers the captured image transmitted by the conference device 5 b-1in the first location to the conference device 5 b-2 in the secondlocation (Step S1704).

The flow of video conference control performed by the conference device5 b-1 serving as the captured-image transmission side when receiving theprocessing information designating the changed image processing methodwill be explained below with reference to FIG. 24. FIG. 24 is aflowchart illustrating the flow of the video conference controlperformed by an information processing apparatus serving as thecaptured-image transmission side when receiving the processinginformation designating the changed image processing method.

The receiving unit 2402 of the conference device 5 b-1 in the firstlocation receives the processing information designating the changedimage processing method from the conference device 5 b-2 in the secondlocation serving as the captured-image reception side during the videoconference (Step S1801).

The image processing unit 2401 determines whether the changed imageprocessing method designated by the received processing information isthe same as the image processing method designated by thepreviously-received processing information (in other words, whether theimage processing according to the changed image processing method iscurrently performed) (Step S1802). If the changed image processingmethod is currently performed (YES at Step S1802), the image processingunit 2401 executes the currently-performed image processing on thecaptured image acquired from the camera 51, and transmits the processedimage to the conference device 5 b-2 in the second location (StepS1803).

In contrast, if the changed image processing method is not currentlyperformed (NO at Step S1802), the image processing unit 2401 performsthe image processing according to the changed image processing methodindicated by the received processing information on the captured imageacquired from the camera 51, and transmits the processed image to theconference device 5 b-2 in the second location (Step S1804).

As described above, according to the conference device 5 b of the secondembodiment, when acquiring a captured image subjected to imageprocessing from another conference device 5 b, which is installed inanother location different from the location of the subject conferencedevice 5 b and which performs the image processing on the captured imageof the another location, and then displaying the acquired captured imageon the display unit 54 of the subject conference device 5 b, the subjectconference device 5 b transmits the processing information designatingthe image processing to be performed on the captured image to theinformation processing apparatus in the another location, and causes theinformation processing apparatus in the another location to perform theimage processing according to the processing information. Therefore, itbecomes possible to cause the information processing apparatus in theanother location to perform, on the captured image of the anotherlocation, the image processing as desired by a user (for example, aconferee) in the location of the subject conference device 5 b.Consequently, it becomes possible to match the captured image that theconferee wants to view and the image actually displayed, so that theuser's usability can be improved.

Furthermore, according to the conference device 5 b of the secondembodiment, when performing the image processing on a captured imageobtained by the imaging unit in the location of the subject conferencedevice 5 b, and then transmitting the captured image subjected to theimage processing to the information processing apparatus that isinstalled in another location different from the location of the subjectconference device 5 b and that displays the captured image on thedisplay unit 54 installed in the another location, the subjectconference device 5 b receives the processing information designatingthe image processing to be performed on the captured image from theanother information processing apparatus and performs the imageprocessing on the captured image according to the received processinginformation. Therefore, the information processing apparatus installedin the another location can display the captured image subjected to thedesired image processing on the display unit 54 of the informationprocessing apparatus installed in the another location. Consequently,when the captured image of the location of the subject device issubjected to the image processing and thereafter transmitted to theinformation processing apparatus installed in another location, itbecomes possible to match the captured image that the conferee in theanother location wants to view and the image actually displayed, so thatthe user's usability can be improved.

In the second embodiment, a case has been explained that the videoconference is conducted between the conference device 5 b-1 in the firstlocation and the conference device 5 b-2 in the second location.However, for example, when the video conference is conducted among theconference devices 5 b-1, 5 b-2, and 5 b-3 in the first to the thirdlocations, the conference device 5 b on the captured-image transmissionside (for example, the conference device 5 b-1 in the first location)transmits the captured image subjected to the image processing to theconference devices 5 b-2 and 5 b-3 in the second and the third locationsvia the server 3. In this case, the conference device 5 b-1 in the firstlocation gives an instruction on which captured image among the capturedimages subjected to different types of image processing is transmittedto each of the conference devices 5 b in the multiple locations to theserver 3. Therefore, the conference device 5 b-1 in the first locationneed not transmit the captured image, so that a processing load can bereduced.

The programs executed by the conference device of the above embodimentsare provided by being recorded in a computer-readable recording medium,such as a compact disc (CD)-ROM, a flexible disk (FD), or a digitalversatile disk, in a computer-installable or a computer-executable fileformat.

Furthermore, the programs executed by the conference device of the aboveembodiments may be stored in a computer connected to a network, such asthe Internet, and provided by being downloaded via the network.Moreover, the programs executed by the conference device of the aboveembodiments may be provided or distributed via a network, such as theInternet.

Furthermore, the programs executed by the conference device of the aboveembodiments may be provided by being incorporated into a ROM or the likein advance.

While, in the above embodiments, an example has been explained that theimage processing apparatus of the present invention is applied to apersonal computer, the present invention may be applied to amultifunction peripheral (MFP) having at least two of a copyingfunction, a printing function, a scanning function, and a facsimilefunction, or an image processing apparatus, such as a copier, a printer,a scanner, or a facsimile machine.

According to an embodiment of the present invention, it is possible todisplay processed image data of the received-image display formatdesignated by operation input by a user from among the pieces of theprocessed image data of the two or more types of transmission imagedisplay formats transmitted by the other conference device. Therefore,it is possible to display a conference screen as desired by a conferee.

Although the invention has been described with respect to specificembodiments for a complete and clear disclosure, the appended claims arenot to be thus limited but are to be construed as embodying allmodifications and alternative constructions that may occur to oneskilled in the art that fairly fall within the basic teaching herein setforth.

What is claimed is:
 1. A video conference system for holding aconference between at least two conference sites by displaying a videoimage that is captured with a video camera on a display device locatedat the conference site, the video conference system comprising: acontrol device located at each of the at least two conference sites, thecontrol device being connected to the video camera and the displaydevice; and a server connected to the control device via a communicationnetwork, the control device comprising: a control terminal configured tocontrol the display device; and an image display format selecting unitconnected to the control terminal, wherein the image display formatincludes at least one format in which a predetermined process isperformed to image data of the video image that is captured by the videocamera, the control device controls transmission and reception of theimage data, and the display device located at a site in which the imagedisplay format is selected by the image display format selection unitdisplays the video image of another site to which the selected imagedisplay format is noticed, with the selected image display format. 2.The video conference system set forth in claim 1, wherein the videocamera includes a wide-angle lens.
 3. The video conference system setforth in claim 1, wherein the predetermined process includes an imagedistortion correction process on the image data.
 4. The video conferencesystem set forth in claim 1, wherein the predetermined process includesenlarging a desired area of the video image.
 5. The video conferencesystem set forth in claim 1, wherein the control terminal located at thesite in which the image display format is selected by the image displayformat selection unit, notices the selected image display format toanother control terminal located at the other site to which the selectedimage display format is noticed, and the other control terminal locatedat the other site where the selected image display format is noticedperforms a process according to the selected image display format on theimage data and provide the processed image data to the server.
 6. Thevideo conference system set forth in claim 5, wherein the servertransmits the image data, from among data received from the othercontrol terminal located at the other site to which the selected imagedisplay format is noticed, with the selected image display format, andtransmits no other data thereof.
 7. The video conference system setforth in claim 1, wherein the control terminal located at the site inwhich the image display format is selected by the image display formatselection unit, notices the selected image display format to the server,and the server receives the image data from the other site to which theselected image display format is noticed, performs a process accordingto the selected image display format on the image data, and provides theprocessed image data to the control terminal located at the site inwhich the image display format is selected by the image display formatselection unit.
 8. The video conference system set forth in claim 7,wherein the server transmits the image data, from among data receivedfrom the other control terminal located at the other site to which theselected image display format is noticed, with the selected imagedisplay format, and transmits no other data thereof.
 9. A method forperforming a video conference system for holding a conference between atleast two conference sites by displaying a video image that is capturedwith a video camera on a display device located at the conference site,the video conference system comprising: a control device located at eachof the at least two conference sites, the control device being connectedto the video camera and the display device; and a server connected tothe control device via a communication network, the control devicecomprising: a control terminal configured to control the display device;and an image display format selecting unit connected to the controlterminal, wherein the image display format includes at least one formatin which a predetermined process is performed to image data of the videoimage that is captured by the video camera, the control device controlstransmission and reception of the image data, the display device locatedat a site in which the image display format is selected by the imagedisplay format selection unit displays the video image of another siteto which the selected image display format is noticed, with the selectedimage display format, the method comprising: selecting a desired imagedisplay format, by the image display format selection unit located atthe site in which the image display format is selected by the imagedisplay format selection unit; noticing the selected image displayformat to another control terminal located at the other site to whichthe selected image display format is noticed; performing on the imagedata a process according to the selected image display format; receivingfrom the server the image data with the selected image display format;and displaying the video image on the display device located at the sitein which the image display format is selected by the image displayformat selection unit.